
AMERICAN INSTITUTES FOR RESEARCH* 


Final Report of the Impacts of the 
National Math + Science Initiative’s 
(NMSI’s) College Readiness Program 
on High School Students’ Outcomes 


Dan Sherman 
Yibing Li 
Marlene Darwin 
Suzanne Taylor 
Mengli Song 


AUGUST 2017 








Final Report of the Impacts of the 
National Math + Science Initiative’s 
(NMSI’s) College Readiness Program 
on High School Students’ Outcomes 


August 2017 


Dan Sherman 
Yibing Li 
Marlene Darwin 
Suzanne Taylor 
Mengli Song 


HAIR 

AMERICAN INSTITUTES FOR RESEARCH* 

1000 Thomas Jefferson Street NW 
Washington, DC 20007-3835 
202.403.5000 

www.air.org 

Copyright © 2017 American Institutes for Research. All rights reserved. 

2230 08/17 


Abstract 


The National Math + Science Initiative’s (NMSI’s) College Readiness Program (CRP) is an 
established program whose goal is to promote science, technology, engineering, and mathematics 
education in high schools to improve students’ readiness for college. It provides teacher, student, 
and school supports to promote high school students’ success in mathematics, science, and 
English Advanced Placement (AP) courses, with a focus on students who are traditionally 
underrepresented in the targeted AP courses. 

Through a federal Investing in Innovation Fund (i3) validation grant awarded to NMSI in 2011, 
CRP was implemented in a total of 58 high schools in two states—Colorado and Indiana— 
beginning in the 2012-13 school year. American Institutes for Research (AIR) conducted an 
independent evaluation of the impacts of CRP on students’ AP outcomes in these schools for the 
three cohorts of schools that adopted the program in sequential years, using a comparative 
interrupted time series (CITS) design that matched comparison schools to program schools in the 
two states. Overall, schools implementing CRP demonstrated significantly larger increases in the 
share of students taking and passing AP tests in targeted areas relative to comparison schools in 
each of the three cohorts of schools, and the gains in CRP schools were sustained over time. 

Fidelity of program implementation was evaluated using a fidelity matrix approach required as 
part of the National Evaluation of the i3 program, which showed that not all elements of the 
program were implemented with high fidelity. Teachers and students were not always able to 
attend all meetings, and schools did not always meet negotiated enrollment targets. Teacher 
survey data indicated that teachers found the professional development activities provided by 
CRP to be the most helpful support they received under CRP, and students reported that the 
tutoring and special study sessions were the most helpful. Although the program provided 
financial incentives to both teachers and students that were tied to student performance on AP 
tests, these incentives were considered the least important element of the program by both 
teachers and students. 
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Executive Summary 

The National Math + Science Initiative’s (NMSI’s) College Readiness Program (CRP)— 
formerly known as the Advanced Placement Training and Incentive Program—has operated in 
more than 1,000 high schools across 34 states and the District of Columbia since its inception in 
2008. The program’s goal is to promote science, technology, engineering, and mathematics 
education by providing teacher, student, and school supports over a 3-year period to increase 
students’ success in mathematics, science, and English Advanced Placement (AP) courses in 
high schools. The program focuses on students who are traditionally underrepresented in the 
targeted AP courses. 

In 2011, NMSI received a 5-year Investing in Innovation Fund (i3) validation grant awarded by 
the United States Department of Education to implement CRP in Colorado and Indiana. Schools 
were actively recruited to participate in the program under the grant, with the same requirements 
for participation that schools would normally have to meet. The first cohort of 18 schools 
(divided between Colorado and Indiana) began participation in the 2012-13 school year, 
followed by 21 schools in the 2013-14 school year and 19 schools in the 2014-15 school year. 

As part of NMSI’s i3 grant, American Institutes for Research (AIR) conducted an independent 
evaluation of both the impact and the fidelity of implementation of CRP and provided data on 
findings to the National Evaluation of the i3 (NEB) program using required reporting templates. 
This report provides a summary and discussion of results on both impact and implementation 
fidelity and also presents findings from surveys of teachers and students that AIR conducted that 
were not required as part of NEB reporting. 

AIR evaluated the impact of CRP using a comparative interrupted time series (CITS) design. The 
key outcome variables for the school-level intervention were the share of all students (Grades 
10-12) in a school taking AP tests in targeted subjects and the share of all students (Grades 10- 
12) who passed these tests at a level that would typically be considered equivalent to college- 
level coursework. Schools that participated in the program (i.e., treatment schools) in each cohort 
were statistically matched to a set of comparison schools that were of similar size and 
demographics within the same state and had comparable AP performance prior to CRP 
implementation. The impact of CRP on a given student outcome was estimated as the difference 
between treatment schools and comparison schools in the change in the outcome over time, 
before and after the start of CRP implementation. 

The primary confirmatory outcome of the program was the share of Grades 10-12 students in a 
school who passed at least one AP test in the targeted subjects of mathematics, science, or 
English. The program provided supports to schools over a 3-year period, and data were available 
for all 3 implementation years for only the first cohort. The figure below shows the impact of the 
program over time for Cohort 1 schools by year of implementation (beginning in school year 
2012-13), where the impact is shown as the difference between the lines plotted for treatment 
schools and comparison schools. The graph shows that although there was little change in the 
outcome variable for comparison schools over time, the share of all Grades 10-12 students in 
CRP schools who took and passed AP tests increased beginning in the first year of the program, 
and differences were sustained into the second and third years. 
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The differences between program and comparison schools were statistically significant for all 3 
years. Findings were similar for measures of AP performance in different subjects and 
combinations of subjects (mathematics alone, science alone, either math or science, or English 
alone) for this cohort. Additionally, there were similar results for the 1-year and 2-year effects 
within the second cohort and significant (but smaller) first-year effects for the third cohort 
(which tended to include smaller schools). 


Percentage of Students Passing AP Exam in Math/Science/English From 2009-10 to 2014-15 in 
Colorado and Indiana, by Study Group 



The treatment group means for all years and the comparison group means for baseline years were unadjusted 
means. The means for the follow-up years for the comparison group were adjusted means based on the estimated 
difference between the treatment and comparison groups in the deviation from baseline to each follow-up year. 

The evaluation also included analyses of fidelity of program implementation at the school level. 
Fidelity of implementation was evaluated using a matrix approach required as part of the 
National Evaluation of the i3 program, and the overall finding was that not all elements of the 
program were consistently implemented with high fidelity. Most notably, teachers and students 
were not always able to attend all meetings, and schools did not always meet negotiated 
enrollment targets. The approach used to measure fidelity of implementation was relatively 
stringent, in that for most indicators at least 80% of schools were required to meet a target for 
implementation. The targets for schools were either dichotomous (e.g., enrollment targets were 
either met or not) or required that a specified share of individuals participate in a required 
activity (e.g., that 80% of teachers attend summer institutes). 

Survey results for teachers showed that teachers found the professional development activities 
provided by CRP to be the most helpful support they received, and students reported that tutoring 
and special study sessions were most helpful. Although the program provides financial 
incentives to both teachers and students that are tied to test perfonnance, these supports were 
considered to be the least important element of the program. 
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Introduction 


Preparing students for postsecondary success in science, technology, engineering, and 
mathematics (STEM) fields is a national priority. In the United States, job opportunities in 
STEM fields are growing. The Bureau of Labor Statistics has projected that about one million 
STEM-related jobs will be added to the economy and that overall employment in STEM 
professions will grow by about 13% by 2022 (Vilorio, 2014). High school students, however, are 
often unprepared for postsecondary STEM disciplines, potentially preventing students from 
taking advantage of these job opportunities. 

Data for high school students who took the 2016 ACT College Entrance Exam (covering 58% of 
graduating high school seniors) revealed that 41% of the students were considered prepared for 
college-level mathematics and only 36% were prepared for college-level science (ACT, 2016). 
These data showed substantial differences in college readiness by race and ethnicity. Across 
racial groups, 70% of Asian American students and 50% of White students met ACT’s college 
readiness benchmarks for mathematics, whereas only 27% of Hispanic students and 13% of 
African American students met these benchmarks. Similarly, 56% of Asian American students 
and 46% of White students met ACT’s college readiness benchmarks for science, but only 21% 
of Hispanic students and 11% of African American students met these benchmarks. 

One approach to better preparing high school students to attend college and pursue STEM- 
related fields is to provide them with coursework that is equivalent to that taught in college. The 
National Math + Science Initiative (NMSI) developed the College Readiness Program (CRP) 
(fonnerly known as the Advanced Placement Training and Incentive Program—APTIP) to 
support rigorous Advanced Placement (AP) coursework that better prepares students for 
postsecondary education, and postsecondary STEM disciplines in particular. 

CRP provides supports to high schools over a 3-year period, with the goal of increasing the 
number of students who take AP courses in STEM-related areas (including English) and the 
expectation that larger numbers of students will take and attain test scores that many 
postsecondary institutions accept as indicative that the student has taken coursework comparable 
to their offerings (i.e., typically 3 or higher on a 5-point scale). Program supports include training 
of teachers, provision of supplies and equipment to schools, tutoring of students, and provision 
of incentive payments to teachers and students that are tied to AP performance. The program is 
expected to change the culture and the operation of schools in such a way that gains realized over 
the 3 years of support provided by CRP can be sustained. 

As part of the federal Investing in Innovation (i3) Fund established under Section 14007 of the 
American Recovery and Reinvestment Act of 2009, NMSI was awarded an i3 validation grant in 
2011 to implement CRP in approximately 60 high schools in both Colorado and Indiana over 3 
school years (2012-13, 2013-14, and 2014-15). As part of NMSI’s grant application, American 
Institutes for Research (AIR) proposed and subsequently conducted an independent evaluation of 
the program to assess its impact and also the fidelity of implementation by NMSI in the two 
states. This report presents findings from the evaluation about the impact of CRP, as well as 
findings about the fidelity of implementation based on the “fidelity matrix” approach required as 
part of the National Evaluation of the i3 program (NEB). 
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Program Background 

NMSI is a national nonprofit organization launched in 2007 that seeks to increase students’ 
engagement and success in STEM education through several programs, including CRP, which 
focuses on supporting high schools, their teachers, and their students. CRP has since been 
implemented by NMSI in over 1,000 high schools in 34 states. NMSI also has implemented the 
program in schools serving military bases; however, these schools are not included in the current 
evaluation. Schools enter CRP through an application process in which they agree to open AP 
classes to all students, meet performance goals that are set in collaboration with NMSI, and 
implement activities specified by NMSI. 

NMSI supports the implementation and replication of CRP at schools through a structure that 
provides oversight at the national level for overall program and fiscal management. Under the 
NMSI program management structure, the organization forms partnerships with other nonprofit 
organizations in participating states. NMSI refers to these partnerships as NMSI state agents 
(NSAs). Each NSA (one per state) provides oversight for the day-to-day implementation of CRP 
across the state and acts as NMSI’s agent supporting districts, schools, and teachers in the state. 

In each participating state, the NSA provides substantive content support for teachers in program 
schools through three content specialists—one for each of the three core subject areas (science, 
mathematics, and English). The content specialists work closely with teachers, through planning 
and providing professional development, resources, monitoring, consultation, and problem 
solving. Content specialists, in collaboration with the NSA, select lead teachers at the schools 
participating in CRP to arrange and lead internal content area meetings for the AP teachers. 

CRP uses the College Board’s AP program within schools because AP is an established high 
school program widely used across the nation with a rigorous curriculum designed to prepare 
students for college-level coursework (https://www.collegeboard.org/about) . Research shows 
that AP’s college-level coursework standards are related to postsecondary success (College 
Board, 2014; NMSI, 2014). Students who master AP coursework and pass AP exams are 3 times 
more likely to earn a college degree (Morgan & Klaric, 2007). AP also is associated with success 
in specific postsecondary disciplines. Students who take AP mathematics and science courses are 
more likely to earn postsecondary degrees in STEM fields (Tai, Liu, Almarode, & Fan, 2010). 

CRP also uses the AP program because AP exam scores are standardized and widely accepted as 
evidence of college preparedness within a tested subject. At the end of each AP course, students 
take an AP exam that assesses their mastery of the course content. Students earn a score of 1 
through 5 on the AP exams; scores of 3 or above are typically considered passing scores. CRP 
uses these scores to measure college readiness consistently across schools implementing the 
program. Therefore, all CRP schools are expected to offer AP courses, expand their enrollment 
in these courses, and expand AP course offerings as necessary. NMSI provides additional 
support to schools by supplying needed equipment and supplies, including specialized items for 
their coursework such as special calculators and laboratory equipment. 

The next section provides an overview of CRP and its key components, which were the focus of 
the analysis of fidelity of program implementation in this evaluation. 
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Program Description 


CRP expects to meet the goal of increasing student achievement and college readiness in STEM 
disciplines by implementing four key components within schools over 3 school years: 

(1) program management, (2) teacher supports, (3) student supports, and (4) awards, as detailed 
in this section. At the end of the 3 years, schools are expected to maintain the program with their 
own resources, although NMSI continues to make program materials available to them. The 
performance of schools outside of the 3-year window of support was not examined as part of this 
evaluation. 

Program management includes making changes to school culture and putting processes in place 
to implement CRP in schools. NMSI is responsible for overseeing national resources and 
implementing fiscal and program management systems, including the monitoring of the key 
milestones that define CRP. 

To this end, NMSI, in conjunction with the NSAs, trains content specialists who then work with 
schools on-site for one full week in the summer prior to the first year of program 
implementation. During this training, the content specialists leam how to manage and monitor 
each aspect of the program, such as supporting and mentoring AP teachers, and managing unique 
situations, such as different learning styles across teachers and different school cultures. Content 
specialists meet during a 2-day retreat each year to receive further training, discuss lessons 
learned, and collaborate on solutions to concerns. 

Each CRP school also has a designated administrator who is either a school administrator or 
someone who works closely with the AP program at the school to implement CRP and serve as a 
school liaison for the NSA. Lead teachers manage the implementation of internal content 
meetings. Under this component, each NSA submits data to NMSI three times a year on AP 
course offerings, AP course enrollment and demographics, and AP exam results, and also 
provides an annual performance report. 

Teacher supports are a critical component of CRP. Teachers are supported through training, 
access to content specialists, and cross-grade-level teams (referred to as “vertical teams”) to help 
align instruction across grades. The purpose of these supports is to increase teachers’ content 
knowledge and improve instruction, especially instruction in AP courses. All AP teachers are 
provided content-based training sessions and workshops on AP standards, and lead teachers who 
provide overall supervision of AP programs within schools additionally participate in summer 
training sessions on monitoring and mentoring for CRP. Lead teachers manage vertical teams to 
connect AP teachers and pre-AP teachers who teach the same subject in a professional learning 
community. 

During the course of the program, AP teachers in CRP schools are expected to attend the 
appropriate College Board Summer Institute each year that they participate in the program. 

These institutes provide a 4-day intensive, content-focused training led by certified College 
Board trainers. The teachers are also provided with online curricular resources that can be used 
in their instruction. 
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Lead teachers and content specialists in mathematics, science, and English who have extensive 
experience in AP courses constantly monitor the AP teachers within schools participating in CRP 
so that they can assist the teachers with instruction (e.g., model lessons) and ensure that each 
teacher is providing the sequencing and rigor needed to achieve significant student growth, as 
shown by improved AP exam scores. This includes ad hoc one-on-one trainings for individual 
teachers. 

To provide additional professional development to teachers, NMSI also provides three 6-hour 
Saturday study sessions during the year, which are taught by master AP teachers. Teachers learn 
both content and pedagogy in these sessions. 

Student supports under CRP include tutoring and Saturday study sessions specific to AP courses. 
Each participating AP teacher is expected to provide students with a minimum of 40 hours of 
afterschool tutoring every year. Students are offered at least four Saturday study sessions in AP 
mathematics, science, and/or English disciplines, which are led by subject-matter experts; these 
sessions include the opportunity to take mock versions of AP examinations. Students also have 
access to online resources related to their homework. 

The College Readiness Program also supports both teachers and students by providing financial 
incentives to teachers and students who participate in the program that are tied to student 
perfonnance on AP exams in targeted subjects. At the end of the school year, teachers are given 
cash payments of $100 for each of their students who passes an AP exam, plus a $1,000 bonus if 
a targeted number of students earn passing scores. Students receive a $100 cash reward for each 
AP test that they pass among the targeted subjects. The awards are meant to motivate students, 
teachers, and administrators to implement CRP with fidelity and create a school culture that 
promotes success in rigorous courses. Additionally, the program covers at least 50% of exam 
fees to reduce barriers to student participation in the AP program. 

Previous Impact Evaluations of CRP 

There have been several evaluations of CRP across multiple states covering its implementation 
by NMSI since 2008, along with evaluations by Jackson (2010, 2014) of an earlier version of the 
program in Texas. Jackson studied the program implemented in nearly 1,500 schools in Texas 
over 10 years (1994-2005). Using a difference-in-differences approach, he analyzed the 
outcomes of student cohorts in schools before and after program implementation in comparison 
with outcomes for the same student cohorts in schools that were not implementing the program. 
Jackson found that after implementation of the program, enrollment in AP courses in program 
schools doubled and the total number of students in school who passed AP exams within 
program schools increased by about 45%. Student performance on the SAT/ACT was also 
significantly higher in program schools than in comparison schools (Jackson, 2010). In addition, 
Jackson (2014) examined longer term outcomes of the program in urban schools and found that 
students who participated in this program were about 8% more likely than comparable students 
to attend college. Students who went on to attend college were also more likely to have higher 
GPAs and to persist in college. 

In an unpublished study of the CRP program in six states that adopted the program beginning in 
2008, Holtzman (2010) found that the program had a significant and positive effect after 1 year 
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on the share of students in schools who took STEM-related AP courses in program schools 
relative to matched comparison schools within the same states. Using a comparative interrupted 
time series (CITS) design, Holtzman found that CRP was associated with a doubling in the share 
of students within a school taking AP tests in targeted subjects and with a 50 percentage point 
increase in the number of students taking and passing the tests. 

Brown and Choi (2015) employed a potential outcomes modeling approach to estimate the 
causal effect of CRP on three cohorts of schools, estimating effects of the program in an over- 
three-year period relative to a control group. They found significant positive effects in the first 
year of the adoption of CRP at program schools in terms of taking AP exams and also passing 
the exams relative to other schools. These effects persisted into the second and third years, with 
an overall increase of approximately 60% in the total number of students passing AP tests in 
targeted subjects. 

Study Design and Research Questions 

The evaluation of CRP under the i3 validation grant consists of two parts: (1) assessment of the 
program’s impact on selected student AP exam outcomes and (2) assessment of the fidelity of 
implementation of CRP. The evaluation included three cohorts of schools that implemented CRP 
in Colorado and Indiana, where “school cohort” was defined by the year the schools began 
implementation, with the first cohort of schools (18 schools) beginning the program in the 2012- 
13 school year, the second cohort (21 schools) in 2013-14, and the third cohort (19 schools) in 
2014-15. The impact evaluation used a CITS design in which schools that NMSI had recruited 
for the program in each state were matched to other public schools in the state based on 
observable characteristics of the schools, including prior performance on AP tests. 

The CITS design measures impact as difference in trend between a treatment group and a 
comparison group over time following adoption of a program such as CRP. Within each state, 
we selected a set of comparison schools that were deemed comparable in terms of observed 
variables to those schools that adopted CRP. The variables used in matching included AP test 
outcomes (passing and testing) at the school level before schools adopted the program, along 
with characteristics of the schools (i.e., enrollment, share of minority enrollment, and economic 
disadvantage). Schools were matched on a cohort-by-cohort basis so that selected schools were 
as comparable as possible at the time that CRP schools adopted the program. 

The specific research questions for the evaluation are as follows: 

1. What were the impacts of CRP on the likelihood that students in the 10th through 12th 
grades within a school took and passed AP exams in targeted subjects (mathematics, 
science, and English) in the two states? 

2. What were the impacts of CRP on the likelihood that students in the 10th through 12th 
grades within a school passed AP exams in targeted subjects (mathematics, science, and 
English) in the two states? 

3. Were all components of the program implemented with fidelity in the two states? 

4. What parts of the program were most useful from the standpoint of participants? 
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The sample of schools used to address the first two questions (the impact study) included all 
schools that adopted CRP under the grant and a matched sample of schools selected as 
comparison schools in the respective states. The implementation study primarily focused on CRP 
schools and used summary measures of implementation that were developed to complete the 
fidelity matrix required to report data to the NEi3. In addition to data on the fidelity of 
implementation, we collected infonnation on how the program was implemented through 
surveys to individuals who coordinated these programs within schools; this provided inputs into 
the fidelity matrix, along with data generated through program records (e.g., attendance at 
summer institutes). We also administered surveys to AP teachers in the schools and a sample of 
students in their classes to gain their perspective on the parts of the program that were the most 
useful. In addition, we also fielded surveys to comparison schools; only about half of the 
comparison schools provided information. Surveys were sent to comparison schools to obtain a 
sense of the “business as usual” at these schools, to examine what types of supports were offered. 

Sample Description 

To use a CITS model, it is necessary to identify a sample of schools that are as comparable as 
possible to program (or treatment) schools. The treatment schools for this evaluation were 
selected by NMSI using standard protocols for recruitment, including the willingness of schools 
to meet specified requirements for participation. In contrast, comparison schools had not been 
recruited by NMSI as part of the i3 grant, so it was not possible to conduct a randomized control 
trial that would assign some recruited schools to a treatment group and the rest to a control group 
that either would not receive treatment or would receive it later. 1 Although this study did not 
randomly assign schools to groups, the CITS design implicitly controls for differences in the 
baseline mean between the treatment and the comparison group. The CITS design has been 
shown to generate estimates that are similar to the RCT benchmark (Somers, Zhu, Jacob, & 
Bloom, 2013). 

The AIR evaluation team identified and recruited a group of comparison schools similar to the 
treatment schools in each of the three cohorts in each state by using the statistical matching 
approach of Mahalanobis distance matching, which identifies potential comparison schools for 
each treatment school based on the overall “closeness” to treatment schools in terms of a 
summary distance measure defined in terms of differences in observed variables (Stuart, 2010). 

Data on school characteristics used for selecting matched comparison schools and for creating 
model variables came from the Common Core of Data (CCD) Public School Universe Survey 
data, collected annually by the National Center for Education Statistics (NCES). The CCD 
contains school demographic characteristics and enrollment data for public elementary and 
secondary schools. The data included school name and location, number of students enrolled in 
each grade level offered at the school, race/ethnicity composition at the school, and number of 
students eligible for the free and reduced-price lunch program. CCD data were linked with the 
College Board AP exam data using school name and school location (such as the high school 


1 NMSI received an i3 scale-up grant in 2015 to conduct a randomized controlled trial (RCT) in which recruited 
schools were randomly assigned to a treatment and a control group that would receive treatment with a 1-year delay. 
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address or zip code) for the corresponding year of the AP exam data. For example, AP exam 
scores from spring 2014 were li nk ed with CCD to the 2013-14 school year. 


The data used to create outcome variables for the impact study were provided by the College 
Board. These data included student-level records for each AP test student during the school year 
or the summer after the school year. The outcome measures used in the analysis were the share 
of students in Grades 10-12 in a school year who had (1) taken an AP test in a targeted subject 
area or who had (2) taken and passed an AP test in a targeted subject area. We only included 
students in Grades 10-12 because in general only students in these grade levels take AP courses. 
If a school had 1,000 students and 100 took an AP test and 50 passed these tests, the school-level 
value of the test-taking variable would be 0.1 (100/1,000) and the school-level value of the test¬ 
passing variable would be 0.05 (50/1,000). 

To create the outcome variables, AP exams were grouped into three subject areas: math, science, 
and English. AP exams in math include exams in Calculus AB, Calculus BC, Computer Science 
A, and Statistics. AP exams in science include exams in Biology, Chemistry, Environmental 
Science, Physics B, Physics C: Electricity & Magnetism, and Physics C: Mechanics (Physics 1 
and Physics 2 were introduced in the 2014-15 school year). AP exams in English include exams 
in English Language & Composition and English Literature & Composition. 

The overall measures of program impact were based on tests across all three areas (math, 
science, and English), but in analysis we also reported findings for math alone, science alone, 
math and science together, and English alone. The primary confirmatory variable in the analysis 
is the share of students in a school who pass an AP test in any of the targeted subjects, in that this 
variable is a general measure of whether students are college ready in terms of having passed at 
least one test in these subjects. Passing rates by subject area provide more-specific infonnation 
on the topics in which students are prepared. The variables related to test taking provide an 
overall measure of participation in AP courses (regardless of whether students pass these tests) 
and provide an indication of the exposure of students to rigorous coursework provided through 
AP courses. 

Comparison schools were selected from public high schools not currently participating in CRP or 
otherwise identified as participating in CRP in a later year. The matching variables included 
school-level averages of AP exam taking and passing rates across the 3 baseline years based on 
data provided by the College Board (e.g., for Cohort 1 schools, the average AP exam taking and 
passing rates in 2009-10, 2010-11, and 2011-12 were used). Additional matching variables 
included three measures of school characteristics from the most recent baseline year—percentage 
of White students, percentage of students eligible for free or reduced-price lunch, and total 
Grades 10-12 enrollment—which were based on the CCD collected by NCES. 

Matching was conducted separately within each state and for each school cohort as the study 
progressed. However, in the report, we present summaries of the baseline school characteristics 
for CRP schools and comparison schools with schools from Colorado and Indiana combined. 
Tables 1 through 3 show the baseline school characteristics separately for the three cohorts of 
schools. Table 4 shows the results for the overall sample. The balance of the overall sample is 
important because some impact analyses relied on data pooled across multiple cohorts. 
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The last column in each table shows the standardized group differences between CRP schools 
and matched comparison schools, which are measures of the baseline balance of the sample. The 
standardized group difference in Grades 10-12 enrollment was calculated as Hedge’s g (group 
mean difference divided by pooled standard deviation); the standardized difference on the other 
matching variables (which were binary) were computed based on the Cox Index (logged odds 
ratio divided by 1.65). 2 A small standardized group difference (e.g., 0.25 or smaller) indicates 
that the CRP schools and the comparison schools were similar on the matching variable. 


Table 1. Baseline Characteristics of CRP and Comparison Schools (CO and IN): Cohort 1 


School Baseline Characteristics 

Mean of CRP 
Schools (N = 18) 

Mean of Matched 
Comparison 
Schools (N = 18) 

Standardized 

Group 

Difference After 
Matching 

% White (2011-12) 

49.36 

52.08 

-0.07 

% eligible for free or reduced-price 
lunch (2011-12) 

53.21 

48.56 

0.11 

Grades 10-12 enrollment (2011-12) 

1,602 

1,563 

0.07 

% taking AP exam in English, 
mathematics, or science (average 
across 2009-10, 2010-11, & 2011-12) 

10.48 

9.95 

0.04 

% passing AP exam in English, 
mathematics, or science (average 
across 2009-10, 2010-11, & 2011-12) 

4.12 

4.23 

-0.02 


Table 2. Baseline Characteristics of CRP and Comparison Schools (CO and IN): Cohort 2 


School Baseline Characteristics 

Mean of CRP 
Schools (N = 21) 

Mean of Matched 
Comparison 
Schools (N = 21) 

Standardized 

Group 

Difference After 
Matching 

% White (2012-13) 

49.63 

58.80 

-0.22 

% eligible for free or reduced-price 
lunch (2012-13) 

50.81 

46.14 

0.11 

Grades 10-12 enrollment (2012-13) 

1,604 

1,562 

0.05 

% taking AP exam in English, 
mathematics, or science (average 
across 2010-11, 2011-12, & 2012-13) 

10.77 

10.29 

0.03 

% passing AP exam in English, 
mathematics, or science (average 
across 2010-11, 2011-12, & 2012-13) 

4.87 

4.61 

0.03 


2 Differences in AP exam taking/passing rates after matching presented in this report vary slightly from the values 
reported in the “NEi3 Data Collection Templates for Reporting Effects and Fidelity of Implementation Findings.” In 
the data collection tables, average values on the baseline characteristics were based on the most recent baseline year 
for the NEi3 tables instead of the averages across the 3 baseline years used to select schools and for balance check. 
We calculated the balance in AP exam taking/passing rates based on the averages across the 3 baseline years 
because it is more consistent with the model specification for the impact analysis. 


American Institutes for Research 


CRP Final Report—8 





















Table 3. Baseline Characteristics of CRP and Comparison Schools (CO and IN): Cohort 3 


School Baseline Characteristics 

Mean of CRP 
Schools (N = 19) 

Mean of Matched 
Comparison 
Schools (N = 19) 

Standardized 

Group 

Difference After 
Matching 

% White (2013-14) 

49.36 

52.08 

-0.07 

% eligible for free or reduced-price 
lunch (2013-14) 

53.21 

48.56 

0.11 

Grades 10-12 enrollment (2013-14) 

875 

820 

0.11 

% taking AP exam in English, 
mathematics, or science (average 
across 2011-12, 2012-13, & 2013-14) 

13.68 

12.70 

0.05 

% passing AP exam in English, 
mathematics, or science (average 
across 2011-12, 2012-13, & 2013-14) 

4.25 

4.59 

-0.05 


Table 4. Baseline Characteristics of CRP and Comparison Schools (CO and IN): All Three Cohorts 


School Baseline Characteristics 

Mean of CRP 
Schools (N = 58) 

Mean of Matched 
Comparison 
Schools (N = 58) 

Standardized 

Group 

Difference After 
Matching 

% White 

53.35 

52.90 

0.02 

% eligible for free or reduced-price 
lunch 

52.53 

50.02 

0.13 

Grades 10-12 enrollment 

1,365 

1319 

0.06 

% taking AP exam in English, 
mathematics, or science (average 
across 3 baseline years) 

12.48 

11.05 

0.24 

% passing AP exam in English, 
mathematics, or science (average 
across 3 baseline years) 

4.53 

4.20 

0.11 


Tables 1 through 3 show that there was some variation across cohorts in variables used in 
matching, both for the program schools and also for the schools selected as comparison schools. 
However, tables show that within cohorts, the absolute value of the standardized differences 
between program and comparison schools were all below 0.25, indicating that groups were 
similar across the variables used for matching. 

Estimation Approach (Impact Models) 

To estimate the impact of CRP, we used a two-level baseline mean CITS model that controlled 
for the mean outcomes at schools across the 3 baseline years. The first cohort of schools 
implemented the program in the 2012-13 school year and was followed through 3 years of 
program implementation. For this first cohort, we computed effects after 1, 2, and 3 years of 
program implementation. The sample was augmented the next year with a second cohort, which 
allowed us to compute both a first-year effect and a second-year effect over the 2 years that they 


American Institutes for Research 


CRP Final Report—9 




















received support under the grant. Finally, for the 2014-15 school year we added a third cohort of 
schools for which we could compute only a 1 -year effect. 

In addition to the effects estimated for individual school cohorts, it was also possible to pool 1- 
year effects across all three cohorts. In addition, we were able to combine 2-year effects across 
the first two cohorts. An advantage of effects pooled across multiple cohorts is that they are 
based on larger sample sizes and thus estimated with greater precision. 

Impact Models for the First School Cohort 

The impact estimate of particular importance for this study is the effect after 3 years of program 
implementation, which could be estimated only for the first cohort in this study. This represents 
the full period over which NMSI implements the program in a school. In this section, we present 
the two-level linear model used to estimate the 3-year impact for the first cohort, where the AP 
exam taking/passing rates in different years (i.e., 3 baseline years and the third follow-up year) 
were nested within schools. The estimation of the 1-year and 2-year impacts for this first cohort 
of schools was based on a similar model, the only difference being that the 1-year impact model 
included the first follow-up year and the 2-year impact model included the second follow-up year 
in the model. The year-specific impacts for the second and third school cohorts were estimated 
using similar models. 

Level 1 (year level): 

Yij = Poj + PijFY^ij + r tj 

where 

• Y^ is the outcome (e.g., the percentage of Grades 10-12 students who passed AP exams 
in math, science, or English) in year i for school j 

• FY3ij is a dummy variable coded 1 for the third follow-up year (defined the same way for 
both CRP schools and comparison schools) and 0 for baseline years 

Level 2 (school level): 

Poj — Too + YoiTREATj + y 02 MEMBERllj + y 03 FRPLllj + y 04 WHITEllj 
+ y q^SU BU RBWj + yo^RURALWj + y^COj + Uqj 

/? 1; - = y 10 + ynTREATj + y 12 MEMBERllj + y 13 FRPLllj + y 14 WHITEllj 
+ y 15 SUBURBllj -\- y^RURALWj -\- y-^-jCOj + U^j 

where 


• TREATj is a dummy indicator for treatment status, coded 1 for CRP schools and 0 for 
comparison schools 

• MEMBER1 lj is the total Grades 10-12 enrollment in school j in 2011-12 (the most 
recent baseline year for Cohort 1), 

• FRPL11 j is the percentage of students eligible for free or reduced-price lunch in 2011-12 
in school j 
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• WHITE11 j is the percentage of White students in 2011-12 in school j 

• SUBURB11 j is a dummy indicator for whether school j was located in a suburban area in 
2011-12 

• RURAL11 j is a dummy indicator for whether school j was located in a rural area in 2011 — 
12 

• CO j is a dummy indicator for state (1 = Colorado, 0 = Indiana) 

The Level 2 coefficient y lt is the estimate of primary interest in this model, which represents the 
difference between treatment schools and comparison schools in the deviation in the outcome 
from baseline to the third follow-up year. 

Pooled 2-Year Impact Model for Cohorts 1 and 2 Schools 

We estimated the impact of CRP after 2 years of implementation based on data pooled across the 
first and second cohorts of schools using the following model: 

Level 1 Model (year level): 

Y ij =p 0j +p lj FY2 ij +r ij 

where FY2ij is a dummy variable indicating whether the year was the second year after the 
adoption of CRP (i.e., “the second follow-up year,” defined the same way for both CRP schools 
and comparison schools): FY2 = 1 for the second follow-up year and 0 for baseline years. 

Level 2 Model (school level): 

/? 0 j — 7oo + YoiTREATj + y 02 MEMBERj + y 03 FRPLj + y 04 WHITEj + y 05 SUBURBj 
+ y 06 RURALj + y 07 COj + y 03 COEIORT2j + U 0 j 

/? 1; - = y 10 + y-nTREATj + y 12 MEMBERj + y 13 FRPLj + y 14 WHITEj + y 15 SUBURBj 
T y 16 RURALj + y 17 COj + Yi^C0HORT2j + U^j 


where TREATj is a dummy indicator for treatment status; MEMBERj, FRPLj, WHITEj, 
SUBURBj, and RURALj are school characteristics at the most recent baseline year (2011-12 for 
Cohort 1 schools, 2012-13 for Cohort 2 schools); CO is an indicator for state (1 = Colorado, 

0 = Indiana); and COHORT2 is an indicator for school cohort (1 = Cohort 2, 0 = Cohort 1). 

The Level 2 coefficient y lx is the estimate of primary interest in this model, which represents the 
overall difference between treatment schools and comparison schools pooled across both cohorts 
in the deviation in the outcome from baseline to the second follow-up year. 

Pooled 1-Year Impact Model Across All Three Cohorts 

We estimated the overall impact of the program after 1 year of implementation across all three 
cohorts using the following model: 
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Level 1 Model (year level): 

Yij = Poj + PijFYUj + nj 

where FY 1 is a dummy variable that coded each year as 0 or 1, indicating whether the year was 
the first year after the adoption of CRP (i.e., “the first follow-up year,” defined the same way for 
both CRP schools and comparison schools): FYlij = 1 for the first follow-up year and 0 for 
baseline years. 

Level 2 Model (school level): 

Poj — Yoo + YoiYREATj + y 02 MEMBERj + y 03 FRPLj + y 04 WHITEj + y os SUBURBj 
+ y 06 RURALj + y 07 COj + y 03 COHORT2j + y 0 c)COHORT3j + U 0 j 

— y 10 + y^TREATj + y 12 MEMBERj + y 13 FRPLj + y 14 WHITEj + y 15 SUBURBj 
+ y 16 RURALj + y-^COj + y^gC 0 H ORT2j + y±gC 0 H 0 RT3j + U-lj 


where TREATj is a dummy indicator for treatment status; MEMBER )■ (Grades 10-12 
enrollment), FRPLj (percent of students eligible for free or reduced-price lunch), WHITE ) 
(percent of White students), SUBURBj (whether or not a school is located in a suburban area, 
with 1 indicating a suburban school and 0 indicating a nonsuburban school), and RURALj (a 
dummy indicator indicating whether or not a school is located in a rural area, with 1 indicating a 
rural school and 0 indicating a suburban or rural school). CO is an indicator for state (1 = 
Colorado, 0 = Indiana). COHORT2 is an indicator for Cohort 2 schools (1 = Cohort 2 schools, 0 
= Cohorts 1 and 3 schools). COHORT3 is an indicator for Cohort 3 schools (1 = Cohort 3 
schools, 0 = Cohorts 1 and 2 schools). 

The Level 2 coefficient y tl is the estimate of primary interest in this model, which represents the 
overall difference between treatment schools and comparison schools across all three cohorts in 
the deviation in the outcome from baseline to the first follow-up year. 

Impact Findings 

In this section, we first present findings for the 3-year impacts for the first cohort of schools. The 
3-year impacts for the first cohort are the key confirmatory findings for the evaluation, in that 
these findings show what impact the 3-year program had after the period of full implementation. 
We then present findings about 1-year impacts pooled across all three cohorts and 2-year impacts 
pooled across the first two cohorts. Additional year-specific impact findings for individual 
cohorts are provided in the appendix. 

In addition to findings for the key outcome measure (i.e., the percentage of Grades 10-12 
students passing at least one AP test in the targeted subjects of math, science, or English), we 
also present findings for the AP test passing rate in each of the three subjects separately and in 
math and science combined. Moreover, we present results for the AP test-taking rate, a measure 
of exposure to AP coursework. For both test-taking and test-passing outcomes, our findings 
show that the largest impacts were in math and science courses—i.e., results combining the three 
subject areas covered by CRP were not dominated by AP outcomes in English. 
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Three-Year Impacts for Cohort 1 Schools 

Table 5 presents findings for the 3-year impacts of CRP for Cohort 1 schools in Colorado and 
Indiana on two sets of AP-related outcomes: percentage of students in a school taking an AP 
exam in targeted subjects and percentage of students passing an AP exam in these subjects. The 
table shows that the percentage of students taking an AP exam in targeted subjects and the 
percentage of students passing an AP exam increased for the treatment schools across subject 
areas but decreased for the comparison schools from baseline to the third year of implementation 
in all subject areas except science (where the AP passing rate increased for both treatment and 
comparison schools). 

The percentage of Grades 10-12 students who took an AP exam in math, science, or English 
during the third year of CRP implementation, for example, increased by 8.64 percentage points 
from the average of the 3 baseline years for the treatment schools but decreased by 2.13 
percentage points for the comparison schools over the same time period. The difference of 10.77 
percentage points was statistically significant at the 0.001 level. In terms of the percentage of 
students passing AP exams in these areas, there was a 3.97 percentage point difference between 
treatment and comparison schools, which was statistically significant at the 0.001 level. 


Table 5. Three-Year Impacts of CRP on Percentage of Students Taking AP Exam and Percentage 
of Students Passing AP Exam in Cohort 1 Schools in Colorado and Indiana, by Subject 


Outcome 

Average Deviation From 
Baseline Mean 

Estimated 

Group 

Difference 

Standard 
Error of 
Difference 

P-Value 


Treatment 

Group 

Comparison 

Group 




Percent Taking AP Exam 






Math/science/English 

8.64 

-2.13 

10.77 

1.69 

0.000*** 

Math/science 

7.06 

-0.99 

8.05 

1.33 

0.000*** 

Math 

3.66 

-0.56 

4.22 

0.72 

0.000*** 

Science 

5.99 

-0.08 

6.07 

1.15 

0.000*** 

English 

3.97 

-1.63 

5.60 

1.41 

0.000*** 

Percent Passing AP Exam 






Math/science/English 

3.12 

-0.85 

3.97 

0.72 

0.000*** 

Math/science 

3.17 

-0.31 

3.48 

0.50 

0.000*** 

Math 

1.76 

-0.17 

1.93 

0.42 

0.000*** 

Science 

2.09 

0.11 

1.98 

0.31 

0.000*** 

English 

1.01 

-0.59 

1.60 

0.64 

0.013* 


Notes. Number of schools = 36 (18 treatment, 18 comparison). 


Average deviation from baseline mean for the treatment group is the unadjusted average deviation across treatment 
schools; average deviation for the comparison group was computed by subtracting the group difference estimate 
based on the impact model from the unadjusted average deviation for the treatment group. 

P-values are based on two-tailed t tests: *p < .05; **p < .01; ***p < .001. 

Figure 1 illustrates the difference between Cohort 1 treatment and comparison schools in both 
states in their change over time in the most important outcome for this study—the school-level 
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percentage of students passing AP exam in math, science, or English. 3 The figure shows clearly 
that the AP exam passing rate for the treatment schools increased substantially from the baseline 
period (2009-10 to 2011-12) to the third year of implementation (2014-15) (by 3.12 percentage 
points), whereas the AP exam passing rate for the comparison schools decreased somewhat (by 
0.85 percentage points) over the same time period. The difference (3.97 percentage points) was 
statistically significant at the 0.001 level. For this cohort, it appears that the impact of the 
program was largely realized in the first year of implementation and maintained into the second 
and third years. 


Figure 1. Percentage of Students Passing AP Exam in Math/Science/English From 2009-10 to 
2014-15 in Colorado and Indiana, by Study Group 



The treatment group means for all years and the comparison group means for baseline years were unadjusted 
means. The means for the follow-up years for the comparison group were adjusted means based on the estimated 
difference between the treatment and comparison groups in the deviation from baseline to each follow-up year. 

Pooled 2-Year Impacts for Cohorts 1 and 2 Schools 

Table 6 presents the results for the 2-year impacts of CRP on AP performance outcomes based 
on data pooled across the first two cohorts of schools in Colorado and Indiana. It shows that the 
pooled 2-year impacts of CRP were positive and significant across all the AP outcomes 
examined. 


3 The estimates of 1-year and 2-year impacts for Cohort 1 are presented in the appendix. 
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Table 6. Pooled 2-Year Impacts of CRP on Percentage of Students Taking AP Exam and 
Percentage of Students Passing AP Exam in Cohorts 1 and 2 Schools in Colorado and Indiana, by 
Subject 


Outcome 

Average Deviation From 
Baseline Mean 

Estimated 

Group 

Difference 

Standard 
Error of 
Difference 

P-Value 

Treatment 

Group 

Comparison 

Group 

Percent Taking AP Exam 






Math/science/English 

8.23 

-0.87 

9.10 

1.12 

0.000*** 

Math/science 

6.33 

-0.94 

7.27 

0.95 

0.000*** 

Math 

3.51 

-0.66 

4.17 

0.56 

0.000*** 

Science 

4.77 

-0.17 

4.94 

0.81 

0.000*** 

English 

4.84 

-0.16 

5.00 

0.92 

0.000*** 

Percent Passing AP Exam 






Math/science/English 

3.46 

-0.11 

3.57 

0.53 

0.000*** 

Math/science 

2.96 

0.04 

2.92 

0.41 

0.000*** 

Math 

1.90 

-0.11 

2.01 

0.32 

0.000*** 

Science 

1.87 

0.15 

1.72 

0.27 

0.000*** 

English 

1.68 

-0.14 

1.82 

0.41 

0.000*** 


Notes. Number of schools = 78 (39 treatment, 39 comparison). 

Average deviation from baseline mean for the treatment group is the unadjusted average deviation across treatment 
schools; average deviation for the comparison group was computed by subtracting the estimated group difference 
from the unadjusted average deviation for the treatment group. 

P-values are based on two-tailed t tests: *p < .05; **p < .01; ***p < .001. 

Figure 2 illustrates the impact of the program on the percentage of students passing AP exam in 
math, science, or English in schools from both Cohorts 1 and 2. It shows that the AP exam 
passing rate was largely stable in both treatment schools and comparison schools during the 
baseline years, then increased substantially in treatment schools but remained stable in 
comparison schools during the first two follow-up years. The difference between the two study 
groups in the change in the AP exam passing rate from baseline years to the second 
implementation year (3.57 percentages points) was statistically significant at the .001 level. 
These findings suggest that the impact of the CRP program persisted through the second 
implementation year based on data pooled across both cohorts. 
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Figure 2. Percentage of Students Passing AP Exam in Math/Science/English From 3 Baseline 
Years (BY1, BY2, and BY3) to First Two Follow-up Years (FY1 and FY2) in Schools in Cohorts 1 
and 2 in Colorado and Indiana, by Study Group 



Notes. Number of schools = 78 (39 treatment, 39 comparison). 

The treatment group means for all years and the comparison group means for baseline years were unadjusted 
means. The means for the follow-up years for the comparison group were adjusted means based on the estimated 
difference between the treatment and comparison groups in the deviation from baseline to each follow-up year. 

Pooled 1-Year Impacts Across All Three Cohorts 

Table 7 presents the results for pooled 1-year impacts of CRP on AP performance outcomes 
across three cohorts of schools in Colorado and Indiana. It shows that CRP’s impacts on all the 
outcomes examined were positive and statistically significant. 


Table 7. Pooled 1-Year Impacts of CRP on Percentage of Students Taking AP Exam and 
Percentage of Students Passing AP Exam in Schools Across All Cohorts in Colorado and Indiana, 
by Subject 


Outcome 

Average Deviation From 
Baseline Mean 

Estimated 

Group 

Difference 

Standard 
Error of 
Difference 

P-Value 

Treatment 

Group 

Comparison 

Group 

Percent Taking AP Exam 






Math/science/English 

7.42 

0.17 

7.25 

0.98 

0.000*** 

Math/science 

5.33 

-0.11 

5.44 

0.73 

0.000*** 

Math 

2.48 

0.04 

2.44 

0.49 

0.000*** 

Science 

4.38 

0.02 

4.36 

0.64 

0.000*** 

English 

4.66 

0.51 

4.15 

0.84 

0.000*** 

Percent Passing AP Exam 






Math/science/English 

2.98 

0.23 

2.75 

0.41 

0.000*** 

Math/science 

2.20 

0.24 

1.96 

0.32 

0.000*** 

Math 

1.15 

0.17 

0.98 

0.23 

0.000*** 
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Outcome 

Average Deviation From 
Baseline Mean 

Estimated 

Group 

Difference 

Standard 
Error of 
Difference 

P-Value 

Treatment 

Group 

Comparison 

Group 

Science 

1.68 

0.22 

1.46 

0.24 

0.000*** 

English 

1.65 

0.12 

1.53 

0.35 

0.000*** 


Notes. Number of schools = 116 (58 treatment, 58 comparison). 


Average deviation from baseline mean for the treatment group is the unadjusted average deviation across treatment 
schools; average deviation for the comparison group was computed by subtracting the estimated group difference 
from the unadjusted average deviation for the treatment group. 

P-values are based on two-tailed t tests: *p < .05; **p < .01; ***p < .001. 

Figure 3 depicts the change in the percentage of students passing the AP exam in math, science, 
or English from the baseline years to the first follow-up year in treatment schools and the 
corresponding change in comparison schools across all three school cohorts. It shows that the AP 
exam passing rate remained stable from the baseline years to the first follow-up year in 
comparison schools but increased substantially in treatment schools. The difference in the 
change between the two groups of schools (2.75 percentage points) was statistically significant at 
the .001 level. 


Figure 3. Change in Percentage of Students Passing AP Exam in Math/Science/English From 3 
Baseline Years (BY1, BY2, and BY3) to First Follow-up Year (FY1) for Schools in All Cohorts in 
Colorado and Indiana, by Study Group 



Notes. Number of schools = 116 (58 treatment, 58 comparison). 

The treatment group means for all years and the comparison group means for baseline years were unadjusted 
means. The means for the follow-up years for the comparison group were adjusted means based on the estimated 
difference between the treatment and comparison groups in the deviation from baseline to each follow-up year. 

The impact findings presented here show that the adoption of CRP was consistently associated 
with gains in both the AP exam-taking rate and the passing rate in treatment schools over time 
from the baseline years, while these rates for comparison schools remained largely unchanged 
over the same time period. Findings about CRP’s impact on the percentage of students passing 
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AP exams are similar to those of earlier research, cited previously, which found that the adoption 
of the program was associated with increases in the number of students passing AP tests of 45% 
(Jackson, 2010) relative to comparison groups for an earlier version of the program to between 
50% for Holtzman (2011) covering CRP in six states and 60% for Brown and Choi (2015) 
covering three recent cohorts of schools across the nation. 

Implementation Findings 

We used two different approaches to examine the implementation of the CRP: a “fidelity matrix” 
approach required by NEB and a survey-based approach. The fidelity matrix approach collects 
information on program implementation in terms of observable indicators related to specific 
elements of a program. CRP has four components, and each has between three and five 
indicators of program performance (shown in Table 8) that were used to assess fidelity of 
implementation based on performance. 


Table 8. Key Components and Indicators of CRP 


Key Component of CRP 

Program Elements/Indicators 

1. Program management 

Reports on problems provided on time 

Student-level data provided by school 

Performance feedback provided by NMSI 

School-level data provided by school 

2. Teacher supports 

Lead teachers identified 

Team meetings held 

Subject-matter experts identified 

Training: summer institute attendance 

Training: Saturday study sessions attendance 

3. Student supports 

Exam fees paid by NMSI 

Equipment and supplies provided as agreed 

Enrollment targets achieved 

4. Awards/incentives provided 

Administrator awards paid 

Teacher awards paid 

Student awards paid 


Under the fidelity matrix approach, the evaluator sets thresholds of perfonnance for each 
indicator to establish whether an element of the program was implemented with fidelity at a 
school, and then results are combined across schools to determine if the element was 
implemented with fidelity at 80% of schools. In some cases implementation could be measured 
on a yes/no basis (all elements of Components 1,3, and 4 as well as the identification of lead 
teachers and state subject-matter experts under Component 2). These elements were assessed as 
being implemented with fidelity if 80% of schools implemented them as planned. 

For those program elements that involved a number of staff participating in an activity (e.g., 
teachers attending summer institutes or Saturday study sessions under Component 2), a school 
was considered to have implemented the program with fidelity if 80% of identified staff in the 
school participated for the required number of sessions (e.g., 4 days of the summer institute; 3 
Saturday sessions). These elements were considered to have been implemented with fidelity 
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across a program school if 80% of schools met the target (e.g., 80% of teachers attending all 
meetings/sessions). 

The evaluation also collected infonnation from a series of annual surveys administered to 
individuals at CRP schools tasked with coordinating the program within their schools, to all AP 
teachers at these schools, and to a sample of students in AP courses. These surveys were 
designed to gather information on participation in activities associated with the program and also 
asked respondents how useful they thought various program supports were. As part of the survey 
process, we also interviewed the individuals responsible for administering the program at the 
schools to obtain additional information on how the program was implemented and to understand 
potential barriers to implementation. 

Fidelity Matrix Results 

The fidelity matrix approach is relatively stringent, in that it sets targets for perfonnance for each 
element of the program and requires that 80% of schools meet targets to be considered as 
implementing the program with fidelity. By the definitions used in the fidelity matrix, CRP was 
not implemented with full fidelity outside of Component 1 (setting up program structure and 
providing data) and Component 4 (making financial rewards). This was a consistent result across 
cohorts and across years. Overall, there were some elements of the program that schools had 
difficulty implementing, mostly related to attendance of teachers at the required number of 
vertical team meetings or teachers and students at Saturday study sessions. 

Within Component 1 (program management), the regional director conducted all visits and 
monitoring activities and the NS As submitted all required student-level data and annual 
perfonnance results to NMSI. For Component 4 (awards/incentives), the NSAs dispersed all 
awards to the administrators, teachers, and students based on AP test results. 

Under Component 2, the lead teachers and NS A content leads in each state fulfilled their roles as 
outlined by NMSI and the NSAs. However, for the summer training, vertical team meetings, and 
Saturday sessions—given that teachers found it difficult to attend sessions outside of the 
regularly scheduled school hours due to family or other responsibilities (such as coaching)— 
most schools did not meet the targets for these elements, and as result they were reported as not 
being implemented with fidelity. 

Under student supports (Component 3), two elements were implemented with fidelity across all 
program schools: paid exam fees and equipment and supplies. In contrast, fewer than 80% of 
schools were able to meet their targets set with NMSI for enrollment increases in AP classes. 

Survey Results 

The study team administered surveys in the spring of each year of CRP implementation to staff 
and students at both CRP schools and comparison schools. The surveys administered at CRP 
schools gathered information on the aspects of the program that teachers and students found 
useful. Additionally, surveys to lead teachers at CRP schools gathered infonnation on whether 
the lead teachers engaged in activities required by the program. Surveys of teachers and students 
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in AP programs at comparison schools were shorter and primarily asked about the extent to 
which they received supports similar to those offered under the CRP. 

At CRP schools, designated administrators were invited to participate; administrators at 
comparison schools who worked closely with the AP program were also invited to take the 
survey. Across the 3 years of the study, the response rate among administrators for CRP schools 
was over 90%. Similarly, in comparison schools the response rates of individuals having lead 
roles in administering activities related to AP programs were 85% across the 3 years of the study. 

The student surveys were originally designed to be administered to a small sample of students— 
approximately 15 to 20 students per school who were taking AP math, science, or English 
courses—at both treatment and comparison schools. However, many schools decided that it 
would be more convenient to invite an entire AP class to participate instead of sampling students 
from each course. As a result, the number of students who responded to the survey at a 
participating school ranged from less than 10 to over 100. 

In CRP schools, teachers and students responded to questions about the usefulness of supports 
offered in all 3 years of the program; in comparison schools, teachers and students responded to 
questions asking about the types of support offered to teachers and students participating in the 
AP program over the same time period. Table 9 shows the survey items that were asked of 
teachers and students about the supports they received. 


Table 9. Survey Items on Teacher and Student Supports 


Respondents 

CRP/Comparison 

Support 

Item 

Response Options 

Teachers 

CRP 

Collaboration 

How would you describe 
the usefulness of the 
collaboration with other 
AP math, science, and 
English teachers? 

Extremely useful, 
somewhat useful, 
slightly useful, not at 
all useful 

Teachers 

CRP 

Professional 

development 

How would you describe 
the usefulness of the 
professional 
development training 
provided through CRP? 

Extremely useful, 
somewhat useful, 
slightly useful, not at 
all useful 

Teachers 

CRP 

Content 

specialist 

support 

How would you describe 
the usefulness of the 
support provided by the 
content specialists? 

Extremely useful, 
somewhat useful, 
slightly useful, not at 
all useful 

Teachers 

CRP 

Incentives 

How important are these 
incentives in 
encouraging your 
teaching of AP math, 
science, or English 
classes? 

Extremely important, 
somewhat important, 
slightly important, not 
at all important 
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Respondents 

CRP/Comparison 

Support 

Item 

Response Options 

Teachers 

Comparison 

Collaboration 

Do you collaborate with 
other AP math, science, 
and English teachers 
(e.g., in a professional 
learning community 
[PLC] or in team 
meetings)? 

Yes, no 

Teachers 

Comparison 

Professional 

development 

Did you receive 
additional content- 
specific professional 
development specifically 
for teaching AP 
classes? 

Yes, no 

Teachers 

Comparison 

Content 

specialist 

support 

How often do you meet 
or communicate 
(virtually or in person) 
with a subject-matter 
expert or experts in your 
field? 

Daily, weekly, 
monthly, other, not at 
all 

Teachers 

Comparison 

Incentives 

What incentives are 
offered to you as an AP 
math, science, or 

English teacher? 

Financial incentives, 
additional PD 
opportunities, 
recertification points, 
leadership position, 
other, none 

Students 

CRP 

Tutoring 

How would you describe 
the usefulness of 
tutoring sessions? 

Extremely useful, 
somewhat useful, 
slightly useful, not at 
all useful 

Students 

CRP 

Tutoring/ 

group 

sessions 

How would you describe 
the usefulness of AP 
student 

conferences/Saturday 
study sessions? 

Extremely useful, 
somewhat useful, 
slightly useful, not at 
all useful 

Students 

CRP 

Incentives 

How important are the 
rewards in encouraging 
you to participate in AP 
math, science, and 

English classes? 

Extremely important, 
somewhat important, 
slightly important, not 
at all important 

Students 

Comparison 

Tutoring 

Are tutoring sessions 
(e.g., one-on-one or 
small group academic 
help either during or 
outside of the school 
day) offered for any of 
your AP math, science, 
or English classes? 

Yes, no 
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Respondents 

CRP/Comparison 

Support 

Item 

Response Options 

Students 

Comparison 

Tutoring/ 

group 

sessions 

Are group academic 
support sessions offered 
(e.g., study sessions or 
tutoring sessions with 
groups of students) for 
any of your AP math, 
science, or English 
courses? 

Yes, no 

Students 

Comparison 

Incentives 

What are the rewards 
offered to encourage 
you to participate in AP 
math, science, and 

English courses? 

Scholarship 
incentives, prizes, 
weighted grades, AP 
course exam fee is 
waived or 

supplemented, other, 
none, do not know 


Teacher supports offered through CRP included collaboration support, professional 
development, support provided by the content specialists, and incentives. Across all 3 years of 
the study, 69% of program teachers who responded to the survey (67% in CO and 71% in IN) 
reported that they collaborated with other AP math, science, and English teachers outside of 
vertical team meetings that were part of CRP. Such collaborations could involve collaboration in 
professional learning communities or in team meetings that were part of teaching AP courses. 
Ninety percent of the teacher respondents (82% in CO and 98% in IN) reported that they 
participated in content-specific professional development offered through CRP. Three quarters of 
the teacher respondents (69% in CO and 81% in IN) reported that they met or communicated 
(virtually or in person) with a content specialist in their AP subject area. 

Figure 4 shows the usefulness of supports or the importance of incentives for program teachers 
who responded that they received these supports separately, by state. In general, teachers across 
both states found the supports useful. In particular, over 90% of teachers found the professional 
development supports either somewhat useful or extremely useful. The supports that teachers 
found to be less important were the incentives. About half of the teachers reported that incentives 
were either slightly important or not at all important. 
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Figure 4. CRP Teachers’ Responses About the Usefulness of Supports, by State 
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The level of usefulness of program supports and the importance of the incentives as reported by 
teachers in CRP schools were similar over time. Figure 5 shows teachers’ average responses to 
questions about the usefulness of the supports offered through the program in each year of 
implementation. On average, teachers reported that the collaboration, professional development, 
and content specialist supports were useful and that the level of usefulness of all these supports 
was very similar across all 3 years of implementation. 


Figure 5. Level of Usefulness of Supports Reported by Teachers in CRP Schools, by Year of 
Implementation 
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Teachers at comparison schools were offered some of the same types of supports for the AP 
program. Overall, about 50% of the teachers (40% in CO and 48% in IN) reported that they 
collaborated with other AP teachers, such as participating in a learning community. Sixty-eight 
percent of the teachers (70% in CO and 66% in IN) reported that they received additional 
content-specific professional development specifically on teaching AP classes. About half of the 
teachers (60% in CO and 44% in IN) reported that they met or communicated with a subject- 
matter expert. Over 80% of the teachers (84% in CO and 82% in IN) reported that no incentives 
were offered to them for teaching AP math, science, or English classes. 

The study team asked students in CRP schools in the third year of the study—when all three 
school cohorts were implementing CRP—to report on the usefulness of the supports provided by 
the program. Students reported on the usefulness of the tutoring supports, the group academic 
sessions (which were Saturday study sessions in CO and AP student conferences in IN), and the 
incentives offered. Seventy-seven percent of the students responded that tutoring sessions were 
offered (66% in CO and 89% in IN). During the third year of the study, 84% of the students 
reported attending one or more Saturday study sessions, and this was similar across both states 
(83% in CO and 86% in IN). 


Figure 6 shows the usefulness of supports or the importance of incentives reported by students in 
CRP schools separately, by state. Overall, students across both states found the supports useful. 
About 90% of the students reported that the tutoring supports and group study sessions were 
somewhat useful or extremely useful (over 85% in CO and over 89% in IN), and 70% of the 
students (71% in CO and 69% in IN) reported that the incentives were somewhat or extremely 
important. 


Figure 6. Level of Usefulness of Supports in Third Year of Implementation Reported by Students 
in CRP Schools, by State 
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Students in comparison schools reported on similar supports in Year 3, including whether or not 
tutoring sessions and group academic supports were offered at their schools and whether 
incentives were provided. In Year 3, about 60% of the students in comparison schools reported 
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that tutoring sessions were offered, and 40% reported that group academic support sessions were 
offered. Further, 70% of students in comparison schools reported that some sort of incentive or 
reward was offered for taking AP classes, such as waiver of the AP course exam fee. 

Discussion 

The impact findings from the independent evaluation of NMSI’s i3 validation grant show a 
consistent pattern across three cohorts of schools, where a significantly larger number of students 
in high schools that adopted the CRP took AP classes and, more importantly, more went on to 
pass AP tests, relative to similar schools not adopting the program. These results held for all 
three target subjects (math, science, and English) taken together (which is the primary outcome 
measure for the evaluation) or individual subjects. The significantly larger gains in CRP schools 
across subjects in the first year of implementation appeared to be sustained over time, in that the 
statistically significant effects in the first year of implementation continued into the second year 
for Cohorts 1 and 2, and into a third year for Cohort 1. 

The evaluation shows that CRP was effective at those schools recruited by NMSI using their 
standard recruitment approaches, relative to other schools within the two states that had similar 
characteristics, including past history in the AP program. The results do not in themselves 
indicate that other schools (including the comparison schools in this evaluation) would 
necessarily achieve the same level of gains if they implemented the program, because schools 
must be motivated and willing to actively participate in the program, which requires the 
commitments of schools, teachers, and students. 

The data collected for the fidelity matrix indicated that CRP was not fully implemented as 
intended largely because teachers were not able to participate in the requisite number of meetings 
for vertical teams (a collaboration activity) and because both teachers and students were not able 
to attend all Saturday study sessions. This led to the conclusion that two key components of the 
program—teacher supports and student supports—were not implemented with fidelity. 

A general issue with the fidelity matrix approach required by the National Evaluation of the i3 
program is that it requires a high degree of fidelity of implementation across most schools for the 
program element (e.g., teacher supports) to be rated as implemented with fidelity. For example, 
at the school level, high fidelity for implementation of summer institutes requires that 80% of all 
AP teachers at the school attend these institutes for the expected 4 days of the program. Even if 
all schools sent all teachers for this training, the fidelity matrix approach would find that this 
element of the program was not implemented with fidelity if two schools had, say, 75% of their 
teachers attending the institute for 3 days rather than 4 days. 

A specific issue with the fidelity matrix approach is that it does not in itself distinguish between 
program elements that are more or less important to the outcomes. For example, program 
incentive payments were made as expected, but neither teachers nor students found them as 
important as other supports. If fidelity information were collected and reported in more detail 
(either by school or tabulated across schools) rather than what is used to create summary 
measures of fidelity under the matrix approach, it would be possible to have a better idea of 
which elements of programs were implemented with high fidelity at how many schools. This 
would provide a better understanding about how well the program was implemented and could 
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also provide school-level implementation measures that could be examined as a potential factor 
affecting outcomes. Data reduction techniques could be used to create summary scales that could 
be developed to identify how implementation of individual elements were correlated with one 
another. 

Although the program may not have implemented all elements with fidelity, the impact findings 
from this evaluation—covering schools in Colorado and Indiana—are similar to those of 
previous evaluations of CRP as implemented in different states and at different time periods. 
Subject to the caveat that schools enter the program on a nonrandom basis, the findings from this 
and other evaluations provide evidence that the adoption of CRP is associated with increases in 
key student outcomes in the first year that can be sustained through the full 3 years of the 
program. 
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Appendix A: Additional Impact Findings 

In this appendix, we present the following impact findings for individual school cohorts: 1-year 
and 2-year impacts for Cohort 1 schools, 1-year and 2-year impacts for Cohort 2 schools, and 1- 
year impacts for Cohort 3 schools. 

One-Year Impacts for Cohort 1 Schools 

Table A-l presents the findings about the 1-year impacts of the CRP program for Cohort 1 
schools on the percentage of Grades 10-12 students taking an AP exam in math, science, and/or 
English, and the percentage of students passing an AP exam in these subjects. As the table 
shows, the percentage of students taking an AP exam and the percentage of students passing an 
AP exam increased for the treatment schools but decreased for the comparison schools from 
baseline to the first year of implementation across subject areas, with the exception only of the 
science AP exam passing rate, which increased for both the treatment and comparison schools. 

The percentage of students in Grades 10-12 who took an AP exam in math, science, or English 
during the first year of CRP implementation, for example, increased by 5.69 percentage points 
from the average of the 3 baseline years for the treatment schools but decreased by almost a full 
(0.99) percentage point for the comparison schools over the same time period. The difference of 
6.67 percentage points was statistically significant at the 0.001 level. The treatment schools also 
significantly outperformed the comparison schools in all the other outcomes examined after 1 
year of implementation. 


Table A-1. One-Year Impacts of CRP on Percentage of Students Taking AP Exam and Percentage 
of Students Passing AP Exam in Cohort 1 Schools in Colorado and Indiana, by Subject 


Outcome 

Average Deviation From 
Baseline Mean 

Estimated 

Group 

Difference 

Standard 
Error of 
Difference 

P-Value 

Treatment 

Group 

Comparison 

Group 

Percent Taking AP Exam 






Math/science/English 

5.69 

-0.99 

6.67 

1.29 

0.000*** 

Math/science 

3.31 

-0.88 

4.19 

0.84 

0.000*** 

Math 

1.78 

-0.51 

2.30 

0.50 

0.000*** 

Science 

3.31 

-0.18 

3.50 

0.79 

0.000*** 

English 

3.00 

-0.48 

3.48 

0.93 

0.000*** 

Percent Passing AP Exam 






Math/science/English 

2.85 

-0.44 

3.29 

0.59 

0.000*** 

Math/science 

2.59 

-0.16 

2.75 

0.53 

0.000*** 

Math 

1.15 

-0.38 

1.53 

0.27 

0.000*** 

Science 

1.97 

0.23 

1.74 

0.41 

0.000*** 

English 

1.31 

-0.13 

1.44 

0.44 

0.000*** 


Notes. 

a. Number of schools = 36 (18 treatment, 18 comparison). 

b. Average deviation from baseline mean for the treatment group is the unadjusted average deviation across 
treatment schools; average deviation for the comparison group was computed by subtracting the group 
difference estimated based on the impact model from the unadjusted average deviation for the treatment group. 

c. P-values are based on two-tailed t tests: *p < .05; **p < .01; ***p < .001. 
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Two-Year Impacts for Cohort 1 Schools 


Table A-2 presents the findings about the 2-year impacts of CRP for Cohort 1 schools in the two 
states on percentage of Grades 10-12 students taking an AP exam in math, science, and/or 
English, and the percentage of students passing an AP exam in these subjects. Overall, the 
findings show that the percentage of student taking an AP exam and the percentage of students 
passing an AP exam increased for the treatment schools across subject areas but decreased for 
the comparison schools from baseline to the second year of implementation in all subject areas 
except science. The percentage of students in Grades 10-12 who took an AP exam in math, 
science, or English during the second year of CRP implementation, for example, increased by 
7.80 percentage points from the average of the 3 baseline years for the treatment schools but 
decreased by 2.29 percentage points for the comparison schools over the same time period. The 
difference of 10.09 percentage points was statistically significant at the 0.001 level. The 
treatment schools also significantly outperfonned the comparison schools in all the other AP 
outcomes examined after 2 years of implementation. 


Table A-2. Two-Year Impacts of CRP on Percentage of Students Taking AP Exam and Percentage 
of Students Passing AP Exam in Cohort 1 Schools in Colorado and Indiana, by Subject 


Outcome 

Average Deviation from 
Baseline Mean 

Estimated 

Group 

Difference 

Standard 
Error of 
Difference 

P-Value 

Treatment 

Group 

Comparison 

Group 

Percent Taking AP Exam 






Math/science/English 

7.80 

-2.29 

10.09 

1.44 

0.000*** 

Math/science 

5.23 

-1.71 

6.94 

1.02 

0.000*** 

Math 

3.35 

-0.82 

4.17 

0.80 

0.000*** 

Science 

3.81 

-0.71 

4.51 

0.84 

0.000*** 

English 

4.47 

-1.25 

5.72 

1.25 

0.000*** 

Percent Passing AP Exam 






Math/science/English 

3.28 

-0.48 

3.76 

0.68 

0.000*** 

Math/science 

2.93 

-0.26 

3.19 

0.50 

0.000*** 

Math 

1.84 

-0.30 

2.14 

0.42 

0.000*** 

Science 

1.90 

0.01 

1.90 

0.37 

0.000*** 

English 

1.58 

-0.30 

1.88 

0.60 

0.002** 


Notes. 

a. Number of schools = 36 (18 treatment, 18 comparison). 

b. Average deviation from baseline mean for the treatment group is the unadjusted average deviation across 
treatment schools; average deviation for the comparison group was computed by subtracting the group 
difference estimated based on the impact model from the unadjusted average deviation for the treatment group. 

c. P-values are based on two-tailed t tests: *p < .05; **p < .01; ***p < .001. 

One-Year Impacts for Cohort 2 Schools 

Table A-3 presents the findings about the 1-year impacts of CRP for Cohort 2 schools in both 
Colorado and Indiana on AP performance outcomes. These findings for Cohort 2 schools are 
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similar to the findings for Cohort 1 schools presented earlier in the appendix: CRP had 
significant positive impacts on all the AP outcomes examined, based on data across the two 
states. Overall, the findings show that the percentage of students taking an AP exam and the 
percentage of students passing an AP exam increased for both the Cohort 2 treatment and 
comparison schools across subject areas, but the differences in increase between CRP and 
comparison schools from baseline to the end of the first year of implementation were significant 
in all subject areas. 


Table A-3. One-Year Impacts of CRP on Percentage of Students Taking AP Exam and Percentage 
of Students Passing AP Exam in Cohort 2 Schools in Colorado and Indiana, by Subject 


Outcome 

Average Deviation From 
Baseline Mean 

Estimated 

Group 

Difference 

Standard 
Error of 
Difference 

P-Value 

Treatment 

Group 

Comparison 

Group 

Percent Taking AP Exam 






Math/science/English 

6.56 

0.71 

5.85 

1.81 

0.001** 

Math/science 

5.31 

0.22 

5.09 

1.35 

0.000*** 

Math 

2.52 

0.22 

2.29 

0.74 

0.002** 

Science 

4.20 

0.09 

4.11 

1.14 

0.000*** 

English 

4.14 

1.11 

3.02 

1.53 

0.048* 

Percent Passing AP Exam 






Math/science/English 

2.74 

0.22 

2.52 

0.57 

0.000*** 

Math/science 

2.13 

0.32 

1.81 

0.45 

0.000*** 

Math 

1.25 

0.15 

1.10 

0.34 

0.001** 

Science 

1.48 

0.22 

1.25 

0.32 

0.000*** 

English 

1.32 

0.09 

1.23 

0.39 

0.002** 


Notes. 

a. Number of schools = 42 (21 treatment, 21 comparison). 

b. Average deviation from baseline mean for the treatment group is the unadjusted average deviation across 
treatment schools; average deviation for the comparison group was computed by subtracting the group 
difference estimated based on the impact model from the unadjusted average deviation for the treatment group. 

c. P-values are based on two-tailed f tests: *p < .05; **p < .01; ***p < .001. 


American Institutes for Research 


CRP Final Report—A-3 





















Two-Year Impacts for Cohort 2 Schools 


Table A-4 presents the findings about the 2-year impacts of CRP for Cohort 2 schools in the two 
states on AP performance outcomes. These findings suggest that CRP had significant positive 
impacts on all the AP outcomes examined after 2 years of implementation, based on data across 
the two states. Similar to the findings about the difference between the two study groups in the 
change in the AP exam passing rate from baseline years to the first follow-up year (2.52 
percentages points), the group difference in the change in the AP exam passing rate from 
baseline to the second follow-up year (3.52 percentage points) was also statistically significant at 
the .001 level. 


Table A-4. Two-Year Impacts of CRP on Percentage of Students Taking AP Exam and Percentage 
of Students Passing AP Exam in Cohort 2 Schools in Colorado and Indiana, by Subject 


Outcome 

Average Deviation From 
Baseline Mean 

Estimated 

Group 

Difference 

Standard 
Error of 
Difference 

P-Value 

Treatment 

Group 

Comparison 

Group 

Percent Taking AP Exam 






Math/science/English 

8.84 

-0.01 

8.85 

1.79 

0.000*** 

Math/science 

7.38 

-0.53 

7.91 

1.66 

0.000*** 

Math 

3.73 

-0.57 

4.30 

0.87 

0.000*** 

Science 

5.67 

0.10 

5.57 

1.43 

0.000*** 

English 

5.47 

0.54 

4.93 

1.31 

0.000*** 

Percent Passing AP Exam 






Math/science/English 

3.69 

0.17 

3.52 

0.87 

0.000*** 

Math/science 

2.99 

0.27 

2.72 

0.67 

0.000*** 

Math 

1.97 

0.01 

1.96 

0.49 

0.000*** 

Science 

1.83 

0.30 

1.53 

0.44 

0.001** 

English 

1.89 

-0.01 

1.90 

0.59 

0.001** 


Notes. 

a. Number of schools = 42 (21 treatment, 21 comparison). 

b. Average deviation from baseline mean for the treatment group is the unadjusted average deviation across 
treatment schools; average deviation for the comparison group was computed by subtracting the group 
difference estimated based on the impact model from the unadjusted average deviation for the treatment group. 

c. P-values are based on two-tailed t tests: *p < .05; **p < .01; ***p < .001. 
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One-Year Impacts for Cohort 3 Schools 


The results for the 1-year impacts of the CRP program for Cohort 3 schools are summarized in 
Table A-5. Overall, the findings are positive and largely consistent with findings about the 1-year 
impacts for the first two cohorts. As the table shows, the percentage of students taking and 
passing an AP exam increased for both the treatment and comparison schools from the baseline 
to the first year of implementation across subject areas. All of the increases were significantly 
larger in the treatment schools than in the comparison schools, except for the percentage passing 
an AP exam in math/science and the percentage passing an AP exam in math, where the 
differences were not significant. The percentage of Grades 10-12 students who took an AP exam 
in math, science, or English during the first year of CRP implementation for these schools, for 
example, increased by 10.03 percentage points from the average of the 3 baseline years for the 
treatment schools but increased only by 0.89 percentage points for the comparison schools over 
the same time period. The difference of 9.14 percentage points was statistically significant at the 
.001 level. 


Table A-5. First-Year Impacts of CRP on Percentage of Students Taking AP Exam and Percentage 
of Students Passing AP Exam in Cohort 3 Schools in Colorado and Indiana, by Subject 


Outcome 

Average Deviation From 
Baseline Mean 

Estimated 

Group 

Difference 

Standard 
Error of 
Difference 

P-Value 

Treatment 

Group 

Comparison 

Group 

Percent Taking AP 

Exam 






Math/science/English 

10.03 

0.89 

9.14 

2.14 

0.000*** 

Math/science 

7.29 

0.55 

6.74 

1.54 

0.000*** 

Math 

3.09 

0.10 

2.99 

1.23 

0.015* 

Science 

5.61 

0.74 

4.87 

1.40 

0.001** 

English 

6.82 

0.88 

5.94 

1.82 

0.001** 

Percent Passing AP 

Exam 






Math/science/English 

3.38 

0.77 

2.61 

1.03 

0.012* 

Math/science 

2.06 

0.80 

1.26 

0.79 

0.111 

Math 

1.02 

0.57 

0.45 

0.58 

0.442 

Science 

1.64 

0.41 

1.23 

0.58 

0.033* 

English 

2.33 

0.30 

2.03 

0.92 

0.028* 


Notes. 

a. Number of schools = 38 (19 treatment, 19 comparison). 

b. Average deviation from baseline mean for the treatment group is the unadjusted average deviation across 
treatment schools; average deviation for the comparison group was computed by subtracting the group 
difference estimated based on the impact model from the unadjusted average deviation for the treatment group. 

c. P-values are based on two-tailed t tests: *p < .05; **p < .01; ***p < .001. 
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